Skip to content

Feat: flexmeasures monitor automations - #2671

Merged
nhoening merged 28 commits into
mainfrom
feat/2519-monitor-automations
Nov 17, 2026
Merged

nhoening merged 28 commits into
mainfrom
feat/2519-monitor-automations

Conversation

@BelhsanHmida

@BelhsanHmida BelhsanHmida commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Description

Adds flexmeasures monitor automations, which emails the host one digest of the automation runs that failed, so broken automations get noticed without watching every asset's Automations page.

Builds on #2670, which has schedule, report and plugin runs record how their jobs ended, so the digest covers every automation type. This PR targets #2670's branch, so its diff shows only the monitoring work; once #2670 merges, it targets main.

  • A run is reported when its execution failed, when it has been executing for longer than a threshold (e.g. its worker died), or when its dispatch failed for good. A forecast run only counts once its AUTOMATION_RUN_MAX_DISPATCH_ATTEMPTS are used up; other types are never retried, so they count after their first failed attempt.
  • Each run is reported once, through a new nullable AutomationRun.alerted_at column, set only after the email is sent. A missed monitoring run is made up for by the next one, and a mail failure leaves the runs for the next one too.
  • Repeated failures are collapsed per automation ("failed 4 times (execution failed: 3, dispatch failed: 1), last at …, last error: …"). No email is sent when nothing failed.
  • Recipients work as for the other monitor commands: FLEXMEASURES_DEFAULT_MONITORING_MAIL_RECIPIENTS, or --recipient.
  • It stays out of flexmeasures jobs run-automations, so a monitoring problem never holds up dispatching.
  • The query (flexmeasures/data/services/automation_monitoring.py) has no type filter: forecast, schedule, report and plugin runs are judged the same way.
  • The migration adds a partial index holding only the runs the monitor has to look at (not reported yet, and failed, running or not fully dispatched).

Decisions worth a look

  • One stuck threshold for all types (answering flexmeasures monitor automations: email the host a digest of failed automation runs #2667's open question). A run counts as stuck once it has been executing for longer than FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER (6 hours by default), overridable per cron line with --stuck-after-minutes. A forecast with many training cycles may legitimately run longer than a schedule, so a per-type threshold could follow, but one value keeps the configuration simple until that need shows up.
  • The first check after upgrading reports the failures already on record. Runs recorded before the upgrade have no alerted_at, so the first digest lists every failed run in the database, collapsed per automation, which may cover months. The alternative is for the migration to mark all existing runs as reported, so that only new failures show up. I did not do that, because an automation that is broken right now would then stay hidden until it fails again.
  • Without any recipient, the command refuses to run. If neither FLEXMEASURES_DEFAULT_MONITORING_MAIL_RECIPIENTS nor --recipient gives an address, it exits with an error, instead of skipping the email the way latest-run and last-seen do. Those commands can afford to skip it, as they re-check each time; this one marks runs as reported, and marking them without telling anyone would hide them for good. The price is that a host whose cron runs this command without any recipient gets an error on every run.

How to test

  • flexmeasures/cli/tests/test_monitor_automations_fresh_db.py: forecast runs driven through the real runner and worker to a failed execution, a stuck execution (reported at 6 hours, not at 5:59) and an exhausted dispatch; which runs count as failed for other types; the threshold's config and CLI override; collapsing per automation; no recipients.
  • test_a_failed_schedule_run_is_reported_by_monitoring, test_a_failed_report_run_is_reported_by_monitoring and test_a_failed_plugin_run_is_reported_by_monitoring: a job of each type fails on the worker, and the digest names it with its error. Each goes red without Feat: automation run outcomes #2670's outcome recording, and without the failed-execution criterion.
  • Each test was seen to fail by breaking what it covers, one change at a time: dropping each of the three failure criteria, the alerted_at filter, the marking of reported runs, the forecast retry exception, the live-claim check, the CLI override, the config setting, the recipient check, the collapsing, and picking the oldest error instead of the latest.

Looks and feel

The host gets one plain-text email per check, and only when something new went wrong. Its subject says how many automations it is about. Each automation gets one short entry: its name, type, asset and organisation, how often it failed and in which way (failed execution, stuck, or failed dispatch), when its latest failed run was due, and the latest error. For example:

Subject: Failed runs of 2 automation(s)

These automations have runs that failed, or that have been executing for longer than 6 hours, which were not reported before:

Automation 7 'Orphan schedule' (scheduling) on asset 6 'toy-battery' of organisation 'Toy Account':
failed 3 times (dispatch failed: 3), last at 2026-10-12 11:34 UTC (run 87)
last error: ValidationError: {'flex-model': {0: {'sensor': ['No sensor found with ID 14.']}}}

Automation 8 'Self-consumption report' (reporting) on asset 5 'toy-building' of organisation 'Toy Account':
failed once (execution failed: 1), last at 2026-10-12 11:34 UTC (run 88)
last error: AssertionError: The resolution of the results (2:00:00) should match that of the output sensor (0:15:00, ID 13).

Each automation's runs are listed on the Automations page of its asset.
We suggest to check the logs.

It goes in bcc to the host's monitoring recipients only: the configured monitoring mail recipients, or whoever is named with the command's recipient option instead (a user ID or an email address). The organisation owning the asset is not emailed; that is a separate sub-issue of #2519. A stuck run reads "stuck" and has no error line. When nothing new failed, the command only prints "All good" and sends nothing.

How to test manually

  1. Prepare. Run this branch with its database migration applied, a local mail catcher (such as MailHog) as the mail server, and a monitoring recipient configured. Have workers running for the forecasting and scheduling queues, and run the automation dispatcher, either from cron once a minute or by hand.

  2. Create a toy battery account. It comes with a battery, a building, solar panels and a day-ahead price sensor that holds no prices yet.

  3. Load prices into the future. On the price sensor's page, open Upload data and upload a CSV with hourly prices from the start of today until a couple of days ahead. Tick Measured instantly, so the prices are stored as realisations, also for the hours still to come, and the scheduler finds prices for its whole planning window.

  4. Add a healthy forecast automation. First give the solar panels some history: on the solar production sensor's page, open Upload data and upload a CSV with a couple of weeks of readings up to today, ticking Measured instantly. Then open the solar panels' asset page, go to its Automations tab and click New automation. Fill in the form as follows:

    • Name: something you will recognise in the email, such as "PV forecast".
    • Type: forecasting, which is selected already.
    • Schedule (cron expression): how often it runs, read in the timezone below. For the test, every minute (five stars) saves waiting; a real one would run, say, every hour on the hour.
    • Recurrence timezone: UTC keeps the times in the email easy to compare; it defaults to the asset's timezone.
    • Existing data source: leave empty. It is for reusing the forecaster and configuration of a data source that already exists.
    • Data generator: leave empty, which uses the default forecaster (TrainPredictPipeline).
    • Data generator config (JSON): leave empty for the forecaster's defaults. This is where its model, training period and regressors would go.
    • Parameters (JSON): what to forecast, and for how long: the sensor to forecast, which is the solar production sensor's ID, and a duration, such as six hours (PT6H). Each run then forecasts the six hours from the moment it was due. Leave out a fixed start, end or prior: the form refuses them, as every run would share that moment.
    • Active: ticked, so the dispatcher picks it up.

    Save it, and it shows up in the list with its next run. With the dispatcher and a forecasting worker running, its runs succeed, so it never appears in the email.

  5. Add a healthy schedule automation. On the battery's Automations page, add a schedule automation for the battery against that price sensor, planning twelve hours ahead from the start of the hour. Let its planning window start on a quarter hour: a window starting at, say, 13:20 does not line up with the price data, and the schedule then fails for want of prices.

  6. Add automations that will fail.

    • A schedule automation like the healthy one, but with a battery sensor you delete afterwards. Its dispatch fails, and a schedule run is never retried.
    • A report or schedule automation that fails on the worker, for instance a schedule priced by a sensor that holds no data.
    • A forecast automation whose target sensor you delete afterwards. Add a sensor to the building, then create a forecast automation through the same form as in step 4, with that sensor's ID in its parameters. Delete the sensor from its page: the page warns that the automation will fail on its next run. Its dispatch then fails, but a forecast run is retried up to five times before it counts.
  7. Let them run. Let the dispatcher and the workers run once, then run the new monitor command. One email arrives in the mail catcher. It lists the deleted-sensor schedule and the automation that failed on the worker, each with its error. It does not list the healthy forecast and schedule, nor the forecast with the deleted sensor, whose dispatch will still be retried.

  8. Check again right away. The command prints "All good" and sends nothing, as each run is reported once.

  9. Repeated failures. Let the dispatcher run a few more minutes without checking, then check once. Each failing automation still takes one entry, now reading "failed N times", with its latest error.

  10. Retries running out. Keep the dispatcher going for about ten minutes. The forecast appears, as a failed dispatch, in the first check after its fifth attempt failed, and not before.

  11. A stuck run (optional). Start a long job, kill its worker mid-job, and check with a stuck threshold of one minute. The run is reported as stuck, and the email says "longer than 1 minute"; with the default threshold, it would only be reported after six hours.

  12. Recipients and a broken mail server (optional). Name a user ID and an extra address as recipients: both receive the email in bcc. Stop the mail catcher and check: the command fails, and the next check, once mail is back, still reports those runs. Without any recipient configured or named, the command refuses to run.

Further Improvements

  • A run whose jobs never start (no worker on the queue), or whose job ended before Feat: automation run outcomes #2670's run recorded it, stays pending and is not reported as stuck yet.
  • A plugin that overrides AutomationHandler.run() to queue its own jobs records no outcomes, so only its failed dispatches are reported.
  • Emailing account admins and consultants, and opting an automation out of monitoring, are separate sub-issues of Monitoring automations #2519.

Related Items

Closes #2667.


Sign-off

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

…s end

Context:
- Only forecast runs followed their jobs, so schedule, report and plugin runs stayed at execution state 'pending' whatever became of their jobs.

Change:
- record_automation_run_jobs records the jobs such a run queued, then takes the outcome of any job that already finished or failed from Redis; a run with no jobs has succeeded.
- The record_outcome_on_automation_run decorator records a job's start, success or failure on its run, found through the job's RQ ID, and wraps make_schedule, the sequential wrap-up job, run_report_job and execute_automation_job.
- Plugin jobs carry automation_run_id in their trigger.
- A schedule that fell back counts as a failed run, as the fallback job is not one of the run's recorded jobs.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- Schedule, report and plugin runs now record how their jobs ended.

Change:
- Schedule runs, single and sequential, succeed once their jobs did and fail with the job's error; a job queued outside a run is not recorded.
- A plugin run records its job's outcome, also when the job finished before the run recorded it.
- Report run tests check the run's outcome: succeeded, nothing to do, and refused.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- Schedule, report and plugin runs now record how their jobs ended, as forecast runs did.

Change:
- Say which jobs each type of run records, that a run without jobs has succeeded, and that a schedule which fell back counts as failed.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- Schedule, report and plugin runs now record how their jobs ended, not only forecast runs.

Change:
- Append PR #2670 to the #2457 entry, and drop "for its forecast jobs", which no longer holds.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- #2519: a monitor command will email a digest of failed automation runs,
  and has to report each failure once, also when a monitoring run is missed.

Change:
- Add a nullable alerted_at column to AutomationRun, with a migration.
- Add a partial index on the runs the monitor looks at
  (not reported yet, and failed, running or not fully dispatched),
  so that its hourly query does not scan the whole run history.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- #2519: a run whose worker died stays "running" for good, so monitoring needs a threshold after which it counts as stuck.

Change:
- Add the setting, defaulting to six hours, well above any job timeout.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…rted yet

Context:
- #2519: monitoring reports runs whose execution failed, which are stuck, or whose dispatch failed for good.

Change:
- Add get_unalerted_failed_automation_runs, with one criterion for every automation type:
  only a forecast run with dispatch attempts to spare is left out, as it will be dispatched again,
  and so is a run that a runner still holds a live claim on.
- Add group_failed_runs_by_automation, collapsing failures per automation with their counts, latest run and latest error.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- #2519: automations fail silently unless someone watches their asset's Automations page.

Change:
- Email one digest of the failed runs to the monitoring recipients (or --recipient),
  collapsed per automation, and mark the runs as reported once it is sent.
- Send nothing when nothing failed, and refuse to run without recipients,
  rather than marking runs as reported that nobody heard of.
- --stuck-after-minutes overrides FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER.
- Kept out of flexmeasures jobs run-automations, so a mail problem never holds up dispatching.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- #2519: cover the new monitor command.

Change:
- Drive forecast runs to a failed execution, a stuck execution and an exhausted dispatch through the real runner and worker,
  and check each is reported once, and not before it counts.
- Check which runs count as failed for other types, the threshold's config and CLI override,
  collapsing per automation, and refusing to run without recipients.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- #2519: new monitor command and setting.

Change:
- Describe what the command reports in the automations feature docs, and point to it from the host's error monitoring page.
- List the command, document the stuck threshold setting, and add a CLI changelog entry.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
@read-the-docs-community

read-the-docs-community Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Context:
- main moved 11 commits on, including #2689, which saves what a data generator returns through generator_results and renames the unchecked-sensor errors to GeneratorWritesUncheckedSensor.

Change:
- Keep the record_outcome_on_automation_run imports next to main's new ones in data/automations.py and services/reporting.py.
- Keep the decorator on make_schedule where main dropped ScheduleWritesUncheckedSensor.
- The report refusal test expects GeneratorWritesUncheckedSensor as the run's error type.
- The failed schedule test breaks the save through save_generator_results, which make_schedule now calls instead of save_to_db.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…omations

Builds the host digest on top of the run outcomes of schedule, report and plugin runs.
Brings in current main as well.
The only conflict was the CLI changelog, where both sides' entries are kept.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…igration

Context:
- Main gained migrations since this branch started, which left two Alembic heads.

Change:
- Make d4b7e9a1c362 revise f7c2e8b1a409, main's head, rather than c5e1a7b94d20.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…ng digest

Context:
- #2670 has schedule, report and plugin runs record how their jobs ended,
  so their failed executions can now reach the host digest.

Change:
- Fail a schedule, a report and a plugin job on the worker, and check the digest names the automation, the failed execution and the job's error.
- The schedule and plugin runs are marked dispatched, as a runner would, so that only their execution can make them count.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…xecution

Context:
- Since #2670, every automation type records how its run's jobs ended.

Change:
- Remove the note from the monitoring section of the automations docs.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
@BelhsanHmida BelhsanHmida mentioned this pull request Oct 12, 2026
3 tasks done
@BelhsanHmida
BelhsanHmida changed the base branch from main to feat/automation-run-outcomes October 12, 2026 01:28
@BelhsanHmida
BelhsanHmida marked this pull request as ready for review October 12, 2026 01:29
Context:
- PR #2671 adds flexmeasures monitor automations.

Change:
- Add one bullet for it to the per-PR automations list.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
BelhsanHmida added a commit that referenced this pull request Oct 12, 2026
…tomations-user-digests

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

# Conflicts:
#	documentation/cli/change_log.rst
#	documentation/features/automations.rst
BelhsanHmida added a commit that referenced this pull request Oct 12, 2026
…tomations-opt-out

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- The v3.0-33 entry on run-stats said only forecast runs record their execution state, which PR #2670 makes untrue.

Change:
- Say execution state and job records are kept for every type of automation.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- main moved 4 commits on, and #2647 changed the rq.job import in test_automations_fresh_db.py.

Change:
- Keep main's `from rq.job import Job, JobStatus` and this branch's `from sqlalchemy import select`.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
@BelhsanHmida BelhsanHmida mentioned this pull request Oct 14, 2026
22 tasks done
Flix6x and others added 2 commits October 20, 2026 12:01
… them

A schedule run now names its jobs after itself and records them before they are queued,
so this branch no longer records them once they are queued, which would record each job twice.
Its jobs report how they ended through the same decorator as before, as they are looked up by their RQ ID.
A sequential schedule's device jobs are keyed by the sensor they schedule, rather than by their position.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <[email protected]>
…ts stored summary

Every queue's exception handler now stores the type and message of a job's error in its meta (#2698),
so a run catching up on a job that already failed records that type, rather than a stand-in type around the last traceback line.
The meta is read afresh, as the run holds the job as it was when queued.
A job without such a summary is still described by its traceback, without a type.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <[email protected]>
Base automatically changed from feat/automation-run-outcomes to main October 20, 2026 21:10
…-monitor-automations

Prepares for merging #2670's squash commit, whose tree is identical to this tip.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Its content is already here from the branch's own lineage, so conflicts take this branch's side.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
The only conflict was the CLI changelog, where main had moved the `show automations` entry up; this branch's `monitor automations` entry is kept.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…igration

Context:
- Main gained a merge migration since the last sync, which left two Alembic heads.

Change:
- Make d4b7e9a1c362 revise bf22b21d0990, main's head, rather than f7c2e8b1a409.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

@nhoening nhoening left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran the suggested manual tests up iuntil step 10 and it works as advertised :)

I have one small refactoring request but then we are good to go.

This also should be tested in the wild to uncover any remaining issues.

Comment thread flexmeasures/cli/monitor.py Resolved
…or CLI

Requested in review, so other code can reuse it.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Otherwise a host's first digest lists every run that ever failed.
The upgrade marks the runs that the monitor would report now, by its rules at this revision, and leaves out runs that may still be retried or finish.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…tomations page

The digest is sent from the CLI, where FlexMeasures does not know its own address, so the new FLEXMEASURES_BASE_URL setting gives it.
Without the setting, the digest carries no links, as before.
The Automations page opens the tab and info panel of the automation named in ?automation=<id>, which show its latest run, its error and its jobs.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
BelhsanHmida added a commit that referenced this pull request Nov 15, 2026
…riable name

Fixes flake8's E741 from the merge of #2671.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…omations

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

# Conflicts:
#	documentation/changelog.rst
#	documentation/cli/change_log.rst
@nhoening
nhoening merged commit 874a2d8 into main Nov 17, 2026
11 checks passed
@nhoening
nhoening deleted the feat/2519-monitor-automations branch November 17, 2026 22:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

flexmeasures monitor automations: email the host a digest of failed automation runs

3 participants